System and method for predictive sports analytics using body-pose information

ABSTRACT

A system is described for analyzing plays of a sporting event based on body-pose information in conjunction with real-world positional tracking data. The system permits segmentation of sporting event plays into discrete time periods and the labeling of appropriate body-pose characteristics for each period. The system allows for comparison and fine-grained analysis of plays with respect to either a series of body-pose vertices that exist in continuous space (i.e., skeleton) or map directly to attributes, using the positional tracking data and play information to account for contextual differences. To enable analysis for the former, the system performs a projection to 3D space, followed by a spatiotemporal alignment step. Through the system, the importance of particular body-pose motions or specific attributes to the success of particular sporting event plays is quantified.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent applicationSer. No. 15/885,668, filed Jan. 31, 2018, which claims priority to U.S.Provisional Patent Application No. 62/452,815 filed Jan. 31, 2017, whichare incorporated by reference in their entireties.

BACKGROUND OF THE INVENTION

Recent years have seen the development and deployment of commercialsports tracking systems for tracking the movement of players, balls, orother objects on a sports playing field. These tracking systems vary intheir operation, and include purely optically-based systems (e.g., usingmultiple cameras), radio-based systems (e.g., using RFID tags embeddedin player equipment), satellite-based systems (e.g., GPS) and hybridsystems. Generally, regardless of the type of tracking system employed,the output of such a system includes the (x, y) location of players,recorded at a high-frame rate. In this manner, the players' behavior hasbeen essentially “digitized” allowing individual game plays to bevisualized via multi-agent trajectories.

However, such systems typically only represent an object's position as asingle point, or average location. These tracking systems, therefore,typically did not represent or account for a player's appearance duringthe contest, such as whether a player was off-balance, or show whether aplayer used good form during a particular maneuver.

Historically, capturing such body-pose information (e.g., the skeletonof a player) within a game situation was unachievable due totechnological limitations. Prior systems required a player to wear amotion-capture suit with reflective markers, so that the player'smovement could be captured in a controlled lab setting with an array ofcameras. However, recent advances in computer vision and machinelearning, along with Graphical Processing Units and “deep learning”architectures have made it possible to estimate 3D body-pose informationfrom a monocular camera view (e.g., broadcast camera view), without theneed for any dedicated motion capture setup, such as systems describedin D. B. M. H. V. Ramakrishna, D. Munoz and Y. Sheikh, “Pose Machines:Articulated Pose Estimation via Inference Machines,” in EuropeanConference on Computer Vision (ECCV), 2014, and V. R. S. Wei and Y.Sheikh, “Convolutional Pose Machines,” in IEEE Conference on ComputerVision and Pattern Recognition (CVPR), 2016, which are herebyincorporated by reference. Regardless, prior systems have been unable todiscern meaningful statistical information for tracking sporting eventsat a fine-grain level based on such body-pose information.

BRIEF SUMMARY OF THE INVENTION

A system is provided for analyzing plays of a sporting event based onbody-pose information in conjunction with real-world positional trackingdata. The system permits the segmentation of sporting event plays intodiscrete time periods and the labeling of appropriate body-posecharacteristics for each period. The system preferably builds aclassifier for plays based on tracking data to account for contextualdifferences, and then compares body-pose attribute values acrossselected sets of plays. Through the system, the importance of particularbody-pose attributes to the success of particular sporting event playscan be quantified.

Although the examples described herein relate specifically to the sportof basketball, the system is not limited to any particular sport, andcan be applied to any sport or domain with body-pose data fine-graintrajectory data, whether from optical tracking data (e.g., SportVU, ormonocular or multi-ocular broadcast video) or wearable devices (e.g.,RFID, GPS).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an architectural environment for a system for sportsanalytics, according to an embodiment of the present system;

FIG. 2 illustrates representations of a sports play through a trackingsystem and with body pose analysis, according to an embodiment of thepresent system;

FIG. 3 is a flow diagram illustrating an overview of the process ofanalyzing sport plays with respect to body-pose information, inaccordance with an embodiment of the present system;

FIG. 4 is a flow diagram illustrating a method of training acomputer-based sport play difficulty classifier, in accordance with anembodiment of the present system;

FIG. 5 is a flow diagram illustrating a method for identifying relevantbody-pose attributes to a set of sports plays, in accordance with anembodiment of the present system;

FIG. 6 is a chart illustrating an exemplary application of an embodimentof the present system to identify relevant body-pose attributes to a setof sports plays;

FIG. 7 is a histogram generated by an embodiment of the present systemto illustrate the identification of relevant body-pose attributes to aset of sports plays; and

FIG. 8 is a histogram generated by an embodiment of the present systemto illustrate the identification of relevant body-pose attributes to aset of sports plays.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present system process body-pose information forsporting events in a manner that enables fine-grained statisticalanalysis of plays and players, aiding the generation of analyticalstatistical predictions for player and team behavior.

A general overview of the context of the system is described withrespect to FIG. 1, in accordance with an embodiment. At a sporting eventtaking place at a venue 110, a tracking system 120 records the motionsof all players on the playing surface, as well as any other objects ofrelevance (e.g., the ball, the referees, etc.). Tracking system 120 canbe an optically-based system using, for example, a plurality of fixedcameras. Alternatively, tracking system 120 can be a radio-based systemusing, for example, RFID tags worn by players or embedded in objects tobe tracked, or tracking system 120 can be another type of system thattracks moving objects. Preferably, tracking system 120 samples andrecords at a high frame rate (e.g., 25 frames per second) so as tominimize quantization, enabling expert humans to select the onset andoffset of plays at precise times (i.e., frame-level), as well asparticular players of interest. Tracking system 120 stores at leastplayer identity and positional information (e.g., (x,y) position) forall players and objects on the playing surface for each frame in a gamefile 140. The game file 140 is preferably augmented with other eventinformation corresponding to the frames, such as game event information(pass, made shot, turnover, etc.) and context information (currentscore, time remaining, etc.), and assembled into data store 150comprising a large number of game files for the given sport (e.g., anentire season of the National Basketball Association games).

In embodiments of the invention, body-pose information is obtained forthe players involved in the sporting event and stored either with theplays in play database 170 or in a separate database. The body-poseinformation for players in the sporting event preferably is representedas points in 3D-space forming a “skeleton” for the player. An example ofsuch a skeleton is shown in FIG. 2. Notably, the body-pose informationfor the event need not be obtained directly at the event site, butinstead may be inferred indirectly from, e.g., a television broadcastusing, for example, known techniques for extracting such body-poseinformation for the event from a monocular camera view.

The use of body-pose information by embodiments permits finer-grainedanalysis for sporting events than has been possible with positionalinformation alone. For example, FIG. 2 shows three frames 202, 204, 206from the video broadcast of a basketball play over a short timeinterval, represented both as a traditional positional representation208, 210, 212 and as 3D body-pose skeletons 214, 216, 218. From thepositional information alone, it appears that the player 220 has a clearopportunity for a shot, but does not shoot the basketball until he isguarded by an opposing player 222. The body-pose information 214, 216,however, suggests that the player has received a poorly placed pass andis recovering, permitting the opponent to close in and force anoff-balance shot.

Turning to FIG. 3, a flow diagram is shown describing the overallprocess of identifying body poses that correlate with successful playsin a sporting event, in accordance with an embodiment. In a firstportion 302 of the process, 3D body pose information for a number ofplays is obtained from video of a sporting event. The body poseinformation may be obtained by known techniques such as those describedabove, and is preferably stored in a 3D skeletal data representation. Ina second phase 304 of the process, a set of body pose attributes ischosen for labeling and investigation. Values are assigned to the chosenattributes for each play or for a subset of plays at the next phase 306.The process then continues by training a play difficulty classifier atphase 308 by using spatial and temporal information for a subset ofplays, to be used with positional and directional information forclassifying the difficulty of a play. Once the attributes have beenassigned and the classifier has been trained, the process permitsfine-grained analysis of plays in the sporting event with respect to the3D body-pose information at phase 310.

In more detail, the set of body pose attributes chosen at phase 304 mayvary based on the sport, the play, or a segment of the play. Thus, theset of body-pose attributes used for one part of a play may differ fromthe set used in another part of the play. For example, in a basketball3-point shot, embodiments preferably segment the play into five parts:a) prior to the player possessing the ball; b) after the playerpossesses the ball but prior to the shot; c) immediately before theshot; d) during the jump and release; and e) after the player lands.Each of these segmented parts of a play may be described with uniquecombinations of attribute values according to body poses. For example,relevant attributes during the “prior to possession” segment couldinclude indicators of a player's movement and direction, while the“during the jump and release” segment could include descriptors of aplayer's jumping and landing feet, and the closeness of the player'slegs. Additionally, overall play attributes may be used, such as for“overall balance” during the entire play. A set of body-pose attributesmeasured for these five parts is shown as follows:

Attribute Description Possible Values Balance OverallBalanced/off-balance Move Prior to Possessing No, set/yes, run/Direction yes, walk/yes, hop Left/ right/forward/backward Pass QualityPrior to Shot Good/too high/too low/ Pump Fake too left/too rightDribble Yes/no Yes/no Move Just Before Shot None/yes, step/yes, run/Direction yes, walk Left/right/ Turn forward/backward None/ Footworkyes, left/yes, right Left foot step/ Right foot step/hop Set Foot Stancevs. Shoulders Aligned/wide/narrow Jump Foot During Shot Left foot/rightfoot/ Legs During Jump both feet Straight up/ Legs During Fall swingfwd/separate Landing Foot Together/wide/split Left/right/both Land FootStance vs. Torso Aligned/behind/ right/left/front Move After ShotNone/left/right/ forward/backwardIn general, the set of body pose attributes reflect the types ofmovement variation that may occur during each segment of a play. Forexample, the attributes might differ for players moving in differentways (running, hopping, and turning) before and after receiving a pass,as opposed to a player who is completely set for his shot as he waitsfor the pass. The attributes describing the landing of a player aftertaking a 3-point shot also can differ, such as by reflecting whether aplayer's feet are front of his torso, whether a player has a widestance, a player's completely-in-line body pose, whether a player landson one leg, etc.

The setting of attributes for plays in phase 304 is preferably performedautomatically through synchronization of the body-pose information withplay information obtained from a tracking system and database such asthat described with respect to FIG. 1. An embodiment identifies relevantplays from a play database (e.g., all three-point attempts from a set ofbasketball games) and then inspects the body-pose informationcorresponding to the identified plays (e.g., a 3-second windowsurrounding the identified timestamp for a play) to infer theappropriate values for the particular attributes. Alternatively oradditionally, the attributes are not defined in advance, but instead areinferred from the database of plays using unsupervised state discovery,such as through techniques described in Hierarchical Aligned ClusterAnalysis for Temporal Clustering of Human Motion, by F. Zhou et al., inIEEE Transactions on Pattern Analysis and Matching Intelligence, v. 35,no. 3, pp. 582-596, 2013, which is herein incorporated by reference.

FIG. 4 shows a flow diagram for using spatial and temporal contextualinformation to train a classifier for the difficulty of a play, asdescribed generally with respect to phase 308 above, in accordance withan embodiment. At step 402, spatial and temporal data is obtained for aset of plays to be used for training the classifier. The data alsoincludes the outcome of a play, such as whether it was successful (e.g.,a made shot) or not (e.g., a missed shot). At step 404, particularspatial and temporal features are selected as the basis for theclassification. These spatial and temporal features can be computed fromdata obtained via a sport tracking system, such as described above, andusing techniques such as those described in D. Cervone, et al.,“POINTWISE: Predicting points and valuing decisions in real time withNBA optical tracking data,” in MIT Sloan Sports Analytics Conference,2014, which is incorporated by reference herein. For example, in abasketball analysis, positional information for a five second timewindow immediately prior to a shot can be used to calculate the timesince last play: free throw, field goal, rebound, dribble, pass, playerpossession, block, and drive. These times, along with player velocities,comprise temporal contextual features. The spatial features, selected tocapture the player configuration at the moment of the shot, can includethe raw player and ball positions, and the angle and distance betweeneach player and the ball. A logistic regression is performed at step 406using any of known methods, to correlate the selected features with thelikelihood of a successful play. Finally, the plays to be analyzed arelabeled according to their likelihood of success. For example, abasketball shot classified with greater than a 53% likelihood of successcan be labeled “Easy”, whereas those shots classified with less than a47% likelihood of success can be labeled “Hard.”

Thus, the result of applying the classifier to plays in the sportingevent is a partitioned set of “easy” and “difficult” plays, which canthen be used to normalize examples to be analyzed with respect to bodypose information so as to minimize other factors. For example, in a3-point shot analysis, if a player is being guarded closely, he may beunbalanced to actually take a difficult shot. This is different from asituation when a player has an open shot, and is unbalanced due to hispoor technique of ball handling. Thus, the analysis system in anembodiment of the present invention normalizes for shot-context.Naturally, open shots can be either made or missed. Similarly, toughshots can sometimes fall and sometimes not. But accounting for thiscontext permits discovery of which attributes make it more or lesslikely for a play to result in success, e.g., for a player to make ashot.

To analyze the contribution of a body pose attributes to a play'ssuccess, an embodiment performs four statistical comparisons, as shownin FIG. 5: a) attribute values in difficult successful plays versusattribute values in difficult failed plays; b) attribute values in easysuccessful plays versus attribute values in easy failed plays; c)attribute values in easy successful plays versus attribute values indifficult successful plays; and d) attribute values in easy unsuccessfulplays versus attribute values in difficult unsuccessful plays. If thereis a statistically significant difference in the presence of anattribute during the first two comparisons, it indicates that theattribute is likely to affect a play's success. If there is astatistically significant difference in the value of an attribute duringthe second two comparisons, then it identifies the attribute as usefulfor determining whether a play is easy or difficult. Preferably, aPearson's chi-squared test is performed for each attribute in the fourcomparisons.

Results from one empirical study are shown in FIG. 6, which showsseveral body-pose attributes having distinctive distributions betweencompared 3-point shot classes (Tough-Made, Tough-Missed, Easy-Made,Easy-Missed). Each of the four comparisons display significantdifferences in at least three body pose attributes (shown in bold),indicating that certain types of body motion correlate with successfulversus unsuccessful shots, regardless of the game context. The datafurther show that the difficulty of a successful play can be predictedfrom body-pose attributes, since there are statistically significantdistributions between successful, easy shots versus successful,difficult shots.

Embodiments further visualize analyzed correlations among body-poseattributes by generating, for example, a histogram. In FIG. 7, a firsthistogram 702 has been generated and displayed to show the comparison ofthe body pose attributes between difficult made 3-point shots (top) anddifficult missed shots (bottom). A second histogram 704 has beengenerated and displayed to show the comparison of the body poseattributes between easy made shots (top) and easy missed shots (bottom).

Embodiments also are used to analyze an individual player's performanceagainst a set of players across various success metrics and contexts.For example, FIG. 8 shows a histogram that has been generated to comparethe body pose attributes during three-point shots taken by a singleplayer (Stephen Curry, top) to the body pose attributes duringthree-point shots taken by everyone else in the league (bottom). Throughthis analysis and visualization, it can be inferred that this playertakes a significantly higher percentage of off-balance shots (the topbar) than other players generally.

In addition to the example of basketball described throughout thisdisclosure, embodiments of the system are not limited to theseparticular sports, and the system is suitable for use in a variety ofother sports, including but not limited to, for example, rugby,volleyball and American football.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

The use of the terms “a” and “an” and “the” and “at least one” andsimilar referents in the context of describing the invention (especiallyin the context of the following claims) are to be construed to coverboth the singular and the plural, unless otherwise indicated herein orclearly contradicted by context. The use of the term “at least one”followed by a list of one or more items (for example, “at least one of Aand B”) is to be construed to mean one item selected from the listeditems (A or B) or any combination of two or more of the listed items (Aand B), unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “having,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted. Recitation of ranges of valuesherein are merely intended to serve as a shorthand method of referringindividually to each separate value falling within the range, unlessotherwise indicated herein, and each separate value is incorporated intothe specification as if it were individually recited herein. All methodsdescribed herein can be performed in any suitable order unless otherwiseindicated herein or otherwise clearly contradicted by context. The useof any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate the inventionand does not pose a limitation on the scope of the invention unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe invention.

Preferred embodiments of this invention are described herein, includingthe best mode known to the inventors for carrying out the invention.Variations of those preferred embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate, and the inventors intend for the invention to be practicedotherwise than as specifically described herein. Accordingly, thisinvention includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the invention unlessotherwise indicated herein or otherwise clearly contradicted by context.

What is claimed:
 1. A method comprising: receiving, by a computingsystem, body pose information for a plurality of plays, the body poseinformation obtained from video data of the plurality of plays, whereineach play of the plurality of plays comprises a plurality of players;for each play of the plurality of plays, segmenting, by the computingsystem, the play into a plurality of components and annotating eachcomponent with a body pose attribute, wherein each body pose attributeis indicative of a type of movement variation occurring in eachcomponent; for each component of the plurality of components, assigning,by the computing system, a value to the associated body pose attribute,the value describing a movement variation corresponding to the type ofmovement variation occurring in the component; and generating, by thecomputing system, a play classifier configured to classify a target playaccording to its likelihood of success by training the play classifierwith the annotated plurality of plays and values assigned to eachcomponent of each play.
 2. The method of claim 1, further comprising:receiving, by the computing system, a selection of a first play from afirst sporting event; and comparing, by the computing system via theplay classifier, the first play to the plurality of plays to determine alikelihood of success of the first play.
 3. The method of claim 2,wherein comparing, by the computing system via the play classifier, thefirst play to the plurality of plays to determine the likelihood ofsuccess of the first play comprises: generating first body poseinformation for each player in the first play.
 4. The method of claim 3,further comprising: comparing the first body pose information for eachplayer in the first play to second body pose information to each playerin the plurality of plays.
 5. The method of claim 3, further comprising:inspecting the first body pose information to infer a value to beassigned to particular body pose attributes of the first play.
 6. Themethod of claim 3, further comprising: generating a histogram comparingthe first body pose information for each player in the first play tosecond body pose information for each player in the plurality of plays.7. The method of claim 1, wherein body pose attributes differ based onone or more of a sport, a type of play, or a segment of a play.
 8. Amethod comprising: receiving, by a computing system, video data for atarget play, the target play comprising a plurality of players;obtaining, by the computing system, body pose information for eachplayer of the plurality of players in the target play; segmenting, bythe computing system, the target play into a plurality of components;annotating, by the computing system, each component with a body poseattribute indicative of a type of movement variation occurring in eachrespective component; assigning, by the computing system, a value toeach body pose attribute, the value describing a movement variationcorresponding to a type of movement variation occurring in eachcomponent; and determining, by the computing system, a contribution of atarget body pose attribute to a success of the target play by comparingthe target body pose attribute in the target play to a similar body poseattribute in a historical play
 9. The method of claim 8, wherein thehistorical play is a historical failed play or a historical successfulplay.
 10. The method of claim 8, further comprising: generating, by thecomputing system, a histogram presenting the body pose attribute duringthe target play to second body pose attributes of other players insimilar plays.
 11. The method of claim 8, further comprising:determining, by the computing system via a play classifier, a likelihoodof success of the target play.
 12. The method of claim 11, wherein theplay classifier is trained to predict a likelihood of success of a givenplay.
 13. The method of claim 8, wherein body pose attributes differbased on one or more of a sport, a type of play, or a segment of a play.14. The method of claim 8, wherein segmenting, by the computing system,the target play into the plurality of components comprises: segmentingthe target play to include a first segment prior to possession, a secondsegment during possession, and a third segment after possession.
 15. Anon-transitory computer-readable medium having one or more sequence ofinstructions stored thereon, which, when executed by a processor, causea computing system to perform operations comprising: receiving, by acomputing system, body pose information for a plurality of plays, thebody pose information obtained from video data of the plurality ofplays, wherein each play of the plurality of plays comprises a pluralityof players; for each play of the plurality of plays, segmenting, by thecomputing system, the play into a plurality of components and annotatingeach component with a body pose attribute, wherein each body poseattribute is indicative of a type of movement variation occurring ineach component; for each component of the plurality of components,assigning, by the computing system, a value to the associated body poseattribute, the value describing a movement variation corresponding tothe type of movement variation occurring in the component; andgenerating, by the computing system, a play classifier configured toclassify a target play according to its likelihood of success bytraining the play classifier with the annotated plurality of plays andvalues assigned to each component of each play.
 16. The non-transitorycomputer-readable medium of claim 15, further comprising: receiving, bythe computing system, a selection of a first play from a first sportingevent; and comparing, by the computing system via the play classifier,the first play to the plurality of plays to determine a likelihood ofsuccess of the first play.
 17. The non-transitory computer-readablemedium of claim 16, wherein comparing, by the computing system via theplay classifier, the first play to the plurality of plays to determinethe likelihood of success of the first play comprises: generating firstbody pose information for each player in the first play.
 18. Thenon-transitory computer-readable medium of claim 17, further comprising:comparing the first body pose information for each player in the firstplay to second body pose information to each player in the plurality ofplays.
 19. The non-transitory computer-readable medium of claim 17,further comprising: inspecting the first body pose information to infera value to be assigned to particular body pose attributes of the firstplay.
 20. The non-transitory computer-readable medium of claim 17,further comprising: generating a histogram comparing the first body poseinformation for each player in the first play to second body poseinformation for each player in the plurality of plays.